Skip to content

Ship only the TensorRT delegate in the ExecuTorch runtime wheel - #4567

Open
shoumikhin wants to merge 63 commits into
pytorch:mainfrom
shoumikhin:executorch-slim-runtime-wheel
Open

shoumikhin wants to merge 63 commits into
pytorch:mainfrom
shoumikhin:executorch-slim-runtime-wheel

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Problem

The runtime wheel bundled a second copy of the ExecuTorch runtime and swapped in its own Python
bindings at import time. Two copies of a runtime in one process can disagree, and whether loading
worked depended on import order. The TensorRT delegate should use the ExecuTorch runtime the user
already installed.

Change

Ship the TensorRT delegate library, its Python registration module, and a CMake package for C++
consumers. Build against the installed ExecuTorch wheel rather than bundling a private runtime, so
there is one runtime in the process.

Keep the delegate on the shared build, repair, artifact and upload path the main wheel already uses.
It keeps its own version, and it pins
the exact ExecuTorch it was built against, down to the label naming the CUDA build, because the
delegate links that runtime's shared library and the two have to be the same build.

Its other dependencies are pinned exactly too, to the versions present when it was built, and it
requires a CUDA runtime. That is narrower than the main wheel, which ships ranges. The delegate is
one compiled library that binds to what it was compiled against, so installing it alongside a
different torch or TensorRT than it saw is the pairing to refuse rather than resolve.

Keep torch_tensorrt.load(..., format="executorch") working, with the entry points the published
package already exposes, so upgrading this wheel alone does not break existing callers. Examples use
ExecuTorch's own Module API. The C++ guidance covers linking the CUDA backend alongside this one and
explains caller-owned outputs for device-resident exports.

Test plan

Ran the ExecuTorch test directory on Linux.

Built and installed the main and companion wheels on Linux aarch64 with CUDA 13.2, then exported and
numerically executed four kinds of program through stock ExecuTorch: TensorRT only, mixed CUDA and
TensorRT in one program, device resident, and externally weighted. Ran the same programs from a C++
consumer built against the installed wheels.

Confirmed the controls fail as expected: a missing delegate and missing external data both error
rather than silently producing wrong output.

Checked the shipped library's dependencies, symbol retention and library search paths, and that a
C++ consumer resolves CUDA, TensorRT and ExecuTorch from the installed wheels.

The x86_64 main wheel passed installation and dependency checks. Full x86_64 companion qualification
is not complete, and one Arm platform-tag check is still open.

@meta-cla meta-cla Bot added the cla signed label Aug 23, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: build system Issues re: Build system component: api [Python] Issues re: Python API component: api [C++] Issues re: C++ API labels Aug 23, 2026
@github-actions
github-actions Bot requested a review from narendasan August 23, 2026 14:13
@shoumikhin
shoumikhin force-pushed the executorch-slim-runtime-wheel branch 3 times, most recently from 44796ff to 3c104cb Compare August 23, 2026 19:00
@shoumikhin
shoumikhin marked this pull request as ready for review August 23, 2026 19:08
@shoumikhin
shoumikhin force-pushed the executorch-slim-runtime-wheel branch from 3c104cb to 4adc20b Compare August 23, 2026 19:27
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 23, 2026
@shoumikhin
shoumikhin force-pushed the executorch-slim-runtime-wheel branch from 4adc20b to 7cac1af Compare August 23, 2026 19:30
@shoumikhin
shoumikhin marked this pull request as draft August 23, 2026 19:31
@shoumikhin
shoumikhin marked this pull request as ready for review August 23, 2026 19:57
@shoumikhin
shoumikhin force-pushed the executorch-slim-runtime-wheel branch 7 times, most recently from ff3379e to a008221 Compare August 24, 2026 17:00
@lanluo-nvidia lanluo-nvidia added this to the v2.15.0 milestone Aug 24, 2026
@lanluo-nvidia lanluo-nvidia added the ci: nightly Run the nightly lane (all tiers incl. llm / kernels / distributed) on every push label Aug 24, 2026
@github-actions
github-actions Bot requested a review from lanluo-nvidia August 24, 2026 17:03
@shoumikhin
shoumikhin force-pushed the executorch-slim-runtime-wheel branch 4 times, most recently from 49b052e to 370c368 Compare August 25, 2026 06:55
The scan for unversioned C++ runtime symbols selected rows typed FUNC or OBJECT. A symbol's type is
no part of the condition it is testing, which is that a reference is undefined and carries no version,
and filtering on it can only lose coverage. Real libraries do carry TLS entries among their undefined
C++ references, and a toolchain emitting NOTYPE would have been skipped too.

Also record what the ExecuTorch suite needs to run locally. It installs the exact pinned nightly,
which exists only on the matching PyTorch channel and never on PyPI, so without the CUDA version set
the documented command fails during setup before a single test runs. CI always sets it and a shell
does not.

Test plan:

Ran the artifact guard suite: 89 passed.

Drove six symbol rows through the filter covering a versioned reference, an unversioned FUNC, OBJECT,
TLS and NOTYPE, and a defined symbol. Only the four undefined unversioned ones are selected, where
previously TLS and NOTYPE were missed.

Measured the type distribution of undefined C++ references in real system libraries and a freshly
built one, which is where the TLS entries showed up.
The companion links one specific ExecuTorch build, so its dependency names an exact version. That
version was published with the part identifying the CUDA build stripped off, which leaves a
requirement that a processor-only build, or a different CUDA build of the same day, satisfies
happily. So the pin looked exact and permitted precisely the pairings it exists to refuse. It now
carries the whole version.

The build-time check that the installed ExecuTorch matches the repository's pin also switched itself
off when the pin file was absent: reading it returned an empty string, and the comparison below only
ran on a non-empty one. A build with no pin to compare against is the case that check exists for, so
it now fails instead. The condition that tested for emptiness goes with it, since it can no longer be
true.

Test plan:

Ran the packaging and artifact guard suites: 133 passed.

Confirmed with the packaging resolver that the previous requirement accepts a processor-only build
and a different CUDA build of the same date, and that the full version refuses all three wrong
variants while accepting the right one.

Added checks that the local part is not stripped again and that the pin reader raises rather than
returning an empty version.
Three checks in the device-resident export example had no test that could reach them. The stub
program handed to them always carried both delegates, its operator table was empty, and the list of
boundary copy operator names was an empty tuple, so nothing could ever match and every rejection
path was unreachable. Disabling all three checks left the suite green.

The stub is now controllable, and there are cases that hand it a program missing one delegate, then
the other, then both, and a program whose operator table still contains a boundary copy. The copy
name list defaults to the real names instead of an empty tuple.

Separately, the test for the runner's device guard replaced the tensor constructor with a stub that
ignored the device argument and reported a CUDA tensor from the test parameter alone. That verified
the guard while saying nothing about whether the script requests a device at all. The stub now
answers from the device it is given.

Test plan:

Ran the example boundary suite: 14 passed, up from 10.

Confirmed each new case fails when the check it covers is disabled in the example, and that the
device test fails when the device argument is dropped from the runner, which previously passed.
Three things made the installed package unusable or misleading.

The in-tree build defines the same target name as this package does, as an interface library over a
private static copy. The config reused whatever target already existed, so a project that pulled in
the in-tree target and then called find_package linked the private copy and never touched the shared
library in the wheel, with only a status message to say so. Reuse is still right when another
subproject ran this same config, so the config now checks that the existing target is the imported
shared library it creates, and stops with an explanation when it is not.

The run path it adds is the absolute path of the machine that configured, and it propagates to every
consumer binary, so a binary built against the wheel does not run anywhere else. That is the right
default for building against an installed wheel, which is what this package is for, and wrong for
anything redistributable, so it is now a switch a consumer can turn off.

The documented prefix path pointed at the ExecuTorch package directory while its config sits two
levels down under share/cmake, so following the instructions gave a package-not-found. It now names
the same path the build itself computes.

Test plan:

Ran the CMake runtime suite: 2 new checks pass, the other 12 still skip without a toolchain.

Reproduced the collision under cmake 4.4.3: an interface library aliased to that target name reports
INTERFACE_LIBRARY, so the new check fires on exactly the case that used to pass silently.

Confirmed the config still parses cleanly, and that the corrected prefix expression evaluates and
produces a share/cmake path.
The delegate registers itself when the package is imported, and that path asks ExecuTorch for its
registered backend names through a private name. An ExecuTorch that does not export it turned an
ordinary import of this package into a bare traceback about a missing symbol, which says nothing
about what to do. It now raises the package's own compatibility error, which already derives from
ImportError, and names the build to install.

The device-resident export wrote its program to the target path and only then checked it, with no
path that removed the file, so a rejected export destroyed a good program and left the reference
file beside it describing something that program no longer was. It now writes beside the target,
checks that, and moves it into place only once every check passes. Each rejection removes the staging
file rather than leaving it behind.

Test plan:

Ran the example boundary suite: 15 passed.

Added a case that starts from an existing program, forces a rejection, and confirms the original
bytes are untouched and no staging file remains. Confirmed it fails when the export writes straight
over the target again.
Loading through the deprecated path accepts an older companion, which registered by swapping in its
own copy of ExecuTorch's bindings and refuses once ExecuTorch's own copy is already loaded. Its
message says to import it earlier, and a caller of this function cannot: the import it collides with
happens inside this library. So the advice pointed at something impossible while the thing that does
work, upgrading the companion, went unmentioned.

That failure is now caught and re-raised naming the upgrade, with the original error kept so the
cause is still visible.

Test plan:

Ran the load compatibility, example boundary, packaging and CMake suites: 95 passed, 12 skipped.

Added a case that stages an older companion whose registration refuses the way the published one
does, and confirms the message names the upgrade rather than the import order.
Three checks on the deprecated entry points read the source as text and never ran anything, so they
could not see the difference between code that works and code that has been commented out. Measured:
commenting out the loader's fallback while leaving the word in place keeps the text check green, and
emptying a shim's body so it no longer registers keeps its check green too.

They are behavioural now. One stages a companion exposing only the older entry point and asserts the
loader calls it. Another imports the package through its own skip-registration switch, since the
native library is not built in a unit test, replaces each shim's call to register so the forwarding is
visible, and asserts both that registration happened and that the deprecation warning was raised.

Test plan:

Ran the shared workflow and load compatibility suites: 65 passed.

Confirmed each new check fails where the text version passed: with the fallback commented out rather
than deleted, and with a shim that keeps its definition and its warning but no longer registers.
The two checks that the deprecated entry points still forward to ExecuTorch looked for a substring in
the file, and a substring is satisfied by a commented-out line. Measured: commenting out one shim's
return leaves the substring check reporting success while the function returns None, which is exactly
the change the check exists to catch.

They read the parsed function now, so a return that is not there is not found. What the forwarding
does at run time is covered separately where the shims are actually called.

Test plan:

Ran the shared workflow, load compatibility, example boundary and packaging suites: 124 passed.

Confirmed directly that with one shim's return commented out the substring check still reports the
text present while the parsed check reports no return value, and that the other shim is unaffected.
The previous attempt at this re-raised RuntimeError unchanged, on the assumption that an error the
build raises itself already stops the install. It does not. During an editable install setuptools
routes a customized build step through its own wrapper, which catches Exception and turns it into a
warning pip hides. RuntimeError is an Exception, and a missing bazel raises exactly that, so the case
this was meant to fix still reported a successful install with no delegate. Only SystemExit escapes,
because it does not derive from Exception, so everything the build raises now becomes one.

The check on this behaviour read the source for the two lines it expected, which the previous version
satisfied while still being swallowed. It now applies the real wrapper to a build that raises and
passes the result through a stand-in for the wrapper setuptools uses.

The CUDA requirement check moves into the build. It ran while the file was merely being read, so it
could refuse a metadata-only build that never compiles anything. The three version lookups beside it
stay: they produce the dependency metadata, which is exactly what such a build asks for.

Test plan:

Ran the packaging and shared workflow suites: 76 passed.

Confirmed that a stand-in for setuptools' wrapper swallows RuntimeError and lets SystemExit through,
and that restoring the previous clause fails the new check on the missing-bazel case while leaving the
other two exception types passing.

Updated the eight pin-refusal cases, which expected the exception that was being swallowed.
Three failures on the last run were all the same mistake, a change made in one place and not in the
others that have to agree with it.

The wheel checker rejected the ExecuTorch requirement it was supposed to validate. Keeping the label
that names the CUDA build was the point of the earlier change, and the checker still compared against
the label-free pin, then separately refused any requirement carrying a label at all. It now expects
the label the build actually linked, confirms the rest of the version still matches the repository
pin, and keeps refusing labels on every other requirement, which do have to resolve anywhere.

The native build still asked for the older Arm platform tag while the guard and the workflow had both
moved to the newer one, so the Arm wheel failed with an unsupported tag. All three name the same tag
now, and a check holds them together.

Selecting the wheel to install by prefix left the pattern unexpanded on a variant whose wheel is named
differently, and pip read it literally. It selects by excluding the companion instead, which is what
that change was for.

Test plan:

Ran the pin, updater, artifact guard and packaging suites: 382 passed.

Confirmed the tag check fails when the native build is put back to the older tag, which is exactly the
state that broke the Arm build. Confirmed the wheel selection keeps the main wheel and drops the
companion against a directory holding both.

Updated the metadata check that asserted the label was stripped, which was the behaviour the earlier
change corrected.
The Arm wheel needs a newer platform floor than x86, because its build container carries no
devtoolset and the C++ runtime symbols the delegate references are not absorbed statically the way
they are on x86. Five places name that floor, and moving it in some of them left the build failing on
whichever one still disagreed: first the native build asked for the old tag, then the wheel checker
expected it, each surfacing only after the previous one was fixed.

The checker derived one floor for both architectures, so it rejected a correctly tagged Arm wheel. It
takes the floor per architecture now. The documentation named the old Arm tag too.

The check that holds these together covered three of the five. It covers all of them, so a floor that
moves in one place and not the rest fails here rather than in a build.

Test plan:

Ran the artifact guard, packaging and pin suites: 322 passed.

Confirmed the check fails when the checker's floor is put back, and again when the documentation is,
each of which is a state that reached CI this round.

Verified against the run in progress that the linting job and the x86_64 build, both failing before,
now pass.
…he label

Two failures in the ExecuTorch suite, both from changes made earlier in this branch.

Moving the CUDA check out of module scope was a mistake and is reverted. It was meant to stop a
metadata-only build being refused, but that check reads only the installed PyTorch, so moving it
addressed nothing the concern was about, and it cost a real property: an unsupported CUDA used to be
reported before anything else was attempted, and afterwards it surfaced later and less clearly. The
version lookups beside it still cannot move, for the reason now recorded there: they produce the
dependency metadata, which is what such a build is asking for.

The case that checks the wheel checker accepts a correct wheel built its ExecuTorch requirement from
the label-free pin, so once the requirement started carrying the label that names the CUDA build, the
checker rejected a wheel that was right. It reads the installed ExecuTorch now, the same source the
checker compares against, and skips where that is absent or is not the pinned build, since neither
says anything about whether a correct wheel is accepted.

Test plan:

Ran the packaging, artifact guard and pin suites: 322 passed.

The ExecuTorch suite needs an installed runtime and cannot be collected locally, which is why both of
these reached CI. Simulated the checker's comparison against all six of its cases instead: the correct
wheel is accepted and each of the five malformed ones is refused, including a label on a requirement
other than ExecuTorch.

Noted while doing that: a local label is only valid with an equality operator, so the unpinned case
has to keep building from the label-free pin.
Anchoring the wheel pattern on a prefix assumed the built wheel is always named torch_tensorrt
followed by a hyphen. The RTX variant is not: it is torch_tensorrt_rtx, so the pattern matched nothing,
stayed unexpanded, and pip read it as a literal filename. That took out every RTX test job on Windows.

The Linux branch already selected by excluding the companion, which is the thing that must not be
installed here. The Windows branch does the same now, so neither depends on guessing what the wheel
is called.

Test plan:

Ran the pin suite's install controls: 5 passed.

Confirmed against a directory holding an RTX-named wheel beside the companion that the exclusion picks
the RTX wheel and drops the companion, where the prefix pattern matched neither and expanded to
nothing.
Restoring the deprecated loader was not enough, and both halves were wrong in a way reading the
source could not show.

The build removed that file from its own output. That cleanup dates from when this package stopped
shipping a Python API, and it kept running after the file came back, so a published wheel had no
forwarder and the released main wheel's import of it failed. It sweeps stale shared objects only now.
Two checks pinned the old behaviour in, including the one that reads the built archive, which is the
only place this was ever visible.

The forwarder also returned the wrong object. The API it replaces returned a wrapper carrying run()
and forward() and raised a not-found error for a missing path, while forwarding to ExecuTorch's own
loader returns neither, so a caller of the published API would have failed on the return value
instead of on the import. It forwards to the main wheel's loader, which is the one that kept that
interface. The deprecation notice named a replacement that returns something different, and now
names one that does not.

Test plan:

Ran the packaging, load compatibility and shared workflow suites: 112 passed.

Confirmed by reading the built archive that it carries the forwarder, and that restoring the removal
fails two checks including that one. Confirmed the original interface from the released source: a
wrapper with run() and forward(), and a not-found error for a missing path.

Confirmed that forwarding to ExecuTorch's loader again fails the new check.
…tions

The suite holding these tests is assigned to the nightly lane only, so a pull request never ran any of
them and nothing they cover could fail before merge. Five of the files need no GPU and no runtime, and
the lint job already runs two of their siblings for exactly that reason, so they join it there. That is
183 checks that previously ran only after landing. The sixth was left out because it imports torch,
which that job does not install.

Two expectations were red at head, both from earlier changes in this branch. The Arm repair test still
required the older platform floor in its parameters and in a later assertion, while the Arm build now
emits the newer one. The supported-CUDA metadata cases stripped the local label from every
requirement, including the ExecuTorch one that now keeps the label naming the build it links.

Test plan:

Ran the five suites in an environment carrying only the lint job's own dependencies: 183 passed, 23
skipped. That is the environment the new step actually gets.

Ran the packaging and artifact guard suites normally: 134 passed.

The suite holding the two stale expectations cannot be collected without an installed runtime, so their
expected values were computed directly and match what the build now produces.
…rs accurate

Four things the shipped package and the compatibility loader got wrong.

The collision guard checked that an existing target of its name was a shared imported library, which
is not the same as checking it is the one this package created. A same-type target pointing at another
file passed, and the consumer would have linked that file believing otherwise. It compares the
location against the library the package found.

The switch controlling the embedded run path was declared with option(), which inside a package
config creates a cache entry in the consumer's project and, depending on the policy in force,
overrides a plain variable they already set, so the documented opt-out could be ignored. It honours
whatever the consumer set and defaults to on.

The compatibility loader rewrote every import failure from registration as "the companion is too
old". Current builds raise their own compatibility error, which derives from ImportError and already
states precisely what is wrong, so an accurate diagnostic was being replaced by a guess. Only the
older failure is rewritten now.

The companion's documentation still described the ExecuTorch requirement as label-free and explained
why a processor-only build could satisfy it, which the requirement no longer allows. It describes what
the build now produces.

Test plan:

Ran the CMake and load compatibility suites: 39 passed, 12 skipped.

Reproduced under a real CMake that a same-name shared target pointing at another file passes the type
check and is caught by the location check, and that a consumer setting the switch off now sees it off.

Rewrote one of my own checks that indexed into the previous structure rather than describing it.
Four things, all cases where a check accepted more than it should have or a change reached wider than
its problem.

The version half of the pin check accepted the target version appearing anywhere in a file, so a
different package that happened to sit at that version could stand in for the ExecuTorch requirement.
It runs through the same pattern that does the rewriting now. The loose form was never needed for
resuming an interrupted run either, because a site a previous run already moved still matches that
pattern.

The resolver's prerelease setting was widened for the whole project inside a commit about workflow
credentials. Every caller that refreshes or checks the lock already passes that setting on the command
line, so the project-wide default was never what decided it, and widening it changed behaviour for
every other resolve for no reason. Reverted.

One check on the deprecated loader required a call that the loader no longer makes, since it now
forwards to the main wheel's loader rather than to ExecuTorch's, so it contradicted the check that
verifies exactly that. It reads the parsed function for a return value instead.

The check the device-resident example exists for, that the output did not come back on the host, had
nothing reaching it: the other test stops at the load. Removing the check left the suite green.

Test plan:

Ran the five suites in an environment carrying only the lint job's dependencies: 186 passed. Ran the
updater and load compatibility suites: 97 passed.

Confirmed the loose version check accepted a file whose requirement had been reformatted away while an
unrelated package sat at the target version, and that the strict form refuses it.

Confirmed the new output case fails when that check is removed from the example, where previously
every test still passed.
Both refused only the case they were written for and let a near neighbour past.

The release tag lookup piped its output straight into a search, so a lookup that failed produced the
same empty result as a release with no tag. A network or credential problem therefore read as "safe to
re-pin", which is the one thing that guard exists to prevent. The lookup and its result are separate
now, and a failed lookup refuses.

The build-time check compared the installed ExecuTorch and the repository pin with the part naming the
build stripped from both, so a processor-only build of the pinned date passed. That build cannot supply
the CUDA runtime the delegate links, and the wheel published afterwards requires a label the build had
never checked. It checks the build as well as the version.

The fixture behind those tests carried a bare version, which is what a processor-only build looks like,
so it could not have caught this. It carries the label a real wheel carries.

Test plan:

Ran the pull request lane's six suites in an environment holding only the lint job's dependencies: 378
passed, 24 skipped.

Added cases for two CUDA labels, a processor-only label, and no label at all: the first two build and
the last two refuse.

Confirmed the track selection still holds across its twelve branch and override cases.
Nine comments and one message described behaviour the code does not have, most of them because an
earlier change in this branch altered the code and left the prose behind. A wrong comment is worse
than none, because a reader trusts it and stops checking.

The two kept entry points were described as an alias and an error. Both are forwarders that warn and
hand back what ExecuTorch provides; neither raises.

The wheel checker was described as splitting a platform tag on dots and matching each part against a
pattern, with an architecture allowlist as the other half of that rule. It has neither. It compares
the whole tag set against one expected tag, which is what rejects a compound tag and a tag for the
wrong architecture alike.

The pin site fallback was described as keeping an older rule of needing at least one coordinate. It
requires whichever single coordinate the file appears to hold.

A docstring explained why an empty return would be wrong, in a function that raises instead.

A path lookup claimed to work before its own package is importable. It runs during that import.

The mismatch message named the pin and the build environment without saying which was which.

The documentation said the build matrix covers two CUDA versions. It covers three, on both
architectures.

The resolver pin's reason is recorded with what was measured and what would allow it to move, since
its own commit message never covered it.

Test plan:

Ran the pull request lane's six suites in an environment holding only the lint job's dependencies: 378
passed, 24 skipped.

Read the matrix script for the CUDA rows it declares, and the checker for how it compares a tag, and
corrected the prose to match each.
…witch

Three findings, one of which needed measuring before it could be answered.

The wheel checker validated the platform tag and never read the object the tag makes a claim about, so
a wheel tagged for one architecture could carry a library built for the other and pass every check.
It reads the machine out of the object's own header now, and there is a case carrying a payload that
contradicts its tag.

Turning off the embedded run path did not do what it said. Measured on Linux: linking an imported
library makes the build record that library's directory as a run path on its own, with no link options
from this package at all, and only skipping the build run path in the consumer's project removes it.
So the switch removed one absolute path and left another. A package config has no business setting
that in someone else's project, so the switch now names what it does not remove instead of implying
the binary is clean.

The delegate install forces the nightly channel by name on release rows too, which reads like an
oversight beside the channel-parameterised installs around it. It is not: the pin is a dated
development build, the test and release channels carry none of it, and resolving through the channel
variable would fail every release build. That is now written where the line is.

Test plan:

Ran the pull request lane's six suites in an environment holding only the lint job's dependencies: 378
passed, 24 skipped.

Read the machine bytes from freshly built objects on an x86 host and an Arm device to confirm the two
values, then confirmed a tag and payload that disagree are refused while matching pairs pass.

Built a consumer against an imported library under CMake with no link options from this package: it
carries the absolute directory as a run path, and skipping the build run path removes it.

Listed both release channels for the pinned version: neither carries it, and the nightly channel
carries ten wheels of it.
The delegate build listed the CUDA channel as an extra index, which leaves the default index in the
set alongside it. A pin naming a version the default index also publishes therefore resolves from
there, and what it publishes is the processor-only build, so the configure step that follows fails on
the missing CUDA extension.

That is reachable rather than theoretical. Moving the pin onto the stable track selects a version the
default index carries, and this file is one of the sites the pin update rewrites, so it is moved to
match. The previous note here argued the version exists nowhere else, which is true only of the
development build the pin names today and stops being true after exactly that move.

ExecuTorch is installed on its own now, with the index restricted to the CUDA channel, so a channel
that cannot supply the pin fails here instead of quietly installing a build the delegate cannot link.
The other build dependencies keep the default index, which is where they come from.

Test plan:

Ran the pin and updater suites: 249 passed.

Reproduced both outcomes for a stable version. With the channel as an extra index, the resolver takes
a non-CUDA wheel from the default index and reports success. With the index restricted to the channel,
it reports that no matching distribution exists.

Confirmed the default index publishes twenty files for that version and none of them carries a CUDA
label, and that the pin update returns no index arguments at all on the stable track.
…ting its dependencies

Restricting the whole resolution to the CUDA channel broke both delegate builds. ExecuTorch declares
thirty dependencies and the channel does not carry them, so nothing could resolve once the default
index was out of the set.

The requirement it was protecting against still stands: naming the channel as an extra index leaves
the default index alongside it, and a pin naming a version that index also publishes resolves to the
processor-only wheel from there, which the delegate cannot link. So the artifact is fetched on its own,
from the channel and nowhere else, and then installed from the file, which leaves its dependencies to
resolve the way every other dependency in this job does.

Test plan:

Ran the pin and updater suites: 249 passed. The pin site in this file still carries exactly one
requirement for the updater to move.

Confirmed the shape of the failure before changing it: both delegate build lanes failed at the step
that installs ExecuTorch, on the commit that restricted the index, and neither had failed there on the
commit before it.

Confirmed ExecuTorch declares thirty dependencies, and that while the channel carries some of them it
does not carry all.
Two attempts at restricting where the delegate build resolves ExecuTorch both failed in ways worse
than the thing they were guarding against. Restricting the whole resolution broke both build lanes,
because ExecuTorch has thirty dependencies the CUDA channel does not carry. Fetching the artifact
separately then broke the checks that drive this step's shell, first on a runner variable they do not
define and then on a second resolver call their stubs do not expect.

The risk is real but it is already covered, one layer down and in a better place. The delegate's own
build reads the installed ExecuTorch's local label and refuses unless it names a CUDA build, so a
processor-only wheel stops the build with a message that says what is wrong, rather than reaching a
configure error about a missing extension. That guard has cases for two CUDA labels, a processor-only
label, and no label at all.

So the install goes back to one command, and the comment records which layer does the refusing.

Test plan:

Ran the six suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 378 passed, 24 skipped. The shared workflow suite, which drives this step's shell
directly, is 32 of those and was the one failing.

Confirmed the aarch64 delegate build succeeded on the previous head, so the resolution itself was
already repaired before this change.
The package config demanded CMake 3.28, on the stated grounds that ExecuTorch's own package rejects
anything older. It does not. ExecuTorch declares 3.19 and drops to a documented variable-based path
below 3.28, because what misbehaves on older CMake is the origin token in its link options. Ours are
an absolute path and the config uses nothing newer than 3.19, so the floor was inherited reasoning
that did not apply here, and it shut out consumers ExecuTorch itself supports.

Declaring any floor in a config is also wrong on its own, because a config runs inside the consumer's
project and overwrote the minimum they had recorded. A consumer that declared 3.20 read 3.28 back
after the call returned.

The floor is now checked rather than declared, at the same 3.19 ExecuTorch uses, with a message naming
both the requirement and the version in use.

The wheel's own native build keeps 3.28 and should: it consumes ExecuTorch's imported targets, which
genuinely need it, and the build system supplies that CMake, so nobody installs it by hand.

Test plan:

Ran the five suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 191 passed, 23 skipped.

Read the floor out of the pinned ExecuTorch wheel's own config rather than assuming it: 3.19, with a
separate check that degrades below 3.28 instead of refusing.

Confirmed under CMake that a consumer declaring 3.20 still reads 3.20 after including the config,
where it previously read 3.28.
The example runner in the wheel could not start. It exited before reaching main with a loader error
naming a TensorRT library, and that library was installed the whole time, in the distribution beside
this one. The CUDA runtime was missing the same way. A program in a wheel is not launched through
Python, so nothing prepares its search path for it, and the only entry it carried reached this
package's own libraries.

The delegate library in the companion wheel already carries the two entries that were missing, which
is why that loads and this did not. The binaries now carry them too. The second example binary had no
run path at all and would have failed the same way.

A check reads the run path entries of both, so a binary added later without them fails here rather
than for whoever installs the wheel.

Test plan:

Ran the five suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 192 passed, 23 skipped.

The failure was found by installing the built wheel on a machine with a GPU and running the binary,
which reported the missing library and exit status 127, and by reading its recorded run path, which
held build sandbox directories and one useful entry.

Confirmed the check fails when either missing entry is taken away again.
Every option the runner takes is a name and value joined by an equals sign, and anything else was
skipped in silence. So a bare path, which is what most people try first, left the model path at its
default and the program loaded a different file than the one it was asked for, then reported success.
That is how it was read as ignoring its argument entirely.

An unrecognised argument now stops the program and prints what the accepted ones are.

Test plan:

Built the argument handling with a compiler on an Arm device and ran four cases: a path given as a
flag is used, a bare path is refused with a message naming it, two flags together are both read, and
an unknown flag is refused.

Ran the five suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 192 passed.
…egrated

Three places decide whether a caller's pointer can go to TensorRT without a copy, and all three
consult one flag. That flag was set from whether the device is an integrated part, which is a
different question from whether it can read pageable host memory.

The two answers differ on real hardware. An H100 reports integrated as zero and pageable access as
one, so the copy-free path was being given up on a device that supports it. In the other direction an
integrated part is not obliged to report pageable access, and there the flag would have bound ordinary
host memory in as though the device could reach it.

The flag now comes from the attribute that answers the question being asked.

Test plan:

Read both attributes on two discrete devices with a compiled program: integrated zero, pageable access
one, host page tables zero. Read them on an integrated device as well, where both are one, which is
why this went unnoticed there.

Compiled the corrected query against the CUDA toolkit and confirmed it reports pageable access one on
the discrete device, where the previous attribute reported zero.
The C++ formatting check failed. One of the two files was clean before this change and my edit broke
it by wrapping a call that fits on one line under this project's column limit. The other needed no
change beyond what this adds.

Test plan:

Ran the project's own pinned formatter over every C++ file this change touches: both clean, and the
one that was already clean before is clean again.

Ran the five suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 192 passed, 23 skipped.

Confirmed both behaviour changes survived the reformat: the device attribute query and the argument
check are unchanged in substance.
… program

A program built without runtime-allocated outputs needs the caller to supply the output buffer. A
caller who does not gets a copy failure, and that failure said three misleading things.

It numbered the output by its position in the whole argument list, so on a one-input engine the first
output reported as "output 1", while the index a caller passes when supplying a buffer counts outputs
from zero. It named no tensor, unlike its sibling messages. And it reported the program as invalid,
when the program is fine and runs correctly once the buffer is there.

It now gives the output's own index and name, says what is missing, and reports an invalid argument.

Test plan:

Checked the index arithmetic against a one-input, two-output engine shape on a device: the first
output now reports as output zero with its binding name, where it previously reported as output one.

Ran the project's own pinned formatter over the file: clean.

Ran the five suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 192 passed, 23 skipped.
A caller could pin the delegate's target device, and the value was taken verbatim. Asking for the
processor was therefore accepted, produced a program, and that program failed at its first
instruction, with a message that said nothing about the request behind it.

This delegate compiles to TensorRT engines, which need a CUDA device, so anything else is refused
where it is asked for, naming what was requested and what is accepted.

Test plan:

Ran the five suites the pull request lane runs, in an environment holding only the lint job's
dependencies: 192 passed, 23 skipped.

Added a case for each of a bare CUDA target, two numbered ones, the processor, and another accelerator:
the first three are accepted and the last two refused. That case needs an installed ExecuTorch, so it
runs in the suite that has one rather than in the pull request lane.

Checked the accept and refuse decision directly for six values, including an empty one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: nightly Run the nightly lane (all tiers incl. llm / kernels / distributed) on every push cla signed component: api [C++] Issues re: C++ API component: api [Python] Issues re: Python API component: build system Issues re: Build system component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants